The present invention relates to improvements in relation to drowsiness detection, specifically in relation to determining the operational state of a driver operating a vehicle, by means of using a multi source input, e.g. by combining information provided by a camera and a vehicle related information source providing information relating to the operation of the vehicle.
Traffic accidents often occur due to driver impairment caused by, for example, drowsiness. In order to prevent accidents caused by driver impairment, it may be vital to provide the driver with a warning message to reestablish the attention of the driver to the surrounding traffic situation, or in a critical situation to advice the driver to take a break or switch to another driver of the vehicle.
Recently, much progress has been made in developing drowsiness detection algorithms that are based on detection of the driver's behavior, including for example using sensor arrangements for monitoring e.g. the eye closure of the driver using a camera, detecting steering wheel operational patterns, etc. By means of using more than one sensor arrangement, a redundancy may be achieved in ease one of the sensor arrangements fails to detect a drowsy driver, and accordingly an improved robustness of the drowsiness detection is made possible.
An exemplary drowsiness detection system is disclosed in US 2003/0151516 A1, where data from two different sensor arrangements are fused using intelligent software algorithms. Specifically, data is provided from a first and a second sensor arrangement, where the first sensor arrangement comprises an array of sensors mounted in the vehicle headliner and seat are used for detecting head movements, and the second sensor arrangement comprises heart rate monitoring sensors placed in the steering wheel, and used for monitoring driver characteristics indicating drowsy driver.
Even though the drowsiness defection system disclosed in US 2003/015156 A1 provides some improvements in relation to both redundancy and robustness of drowsiness detection, it fails to provide a solution suitable for more generally combining data from arbitrary sensor arrangements, possibly having different timelines and/or sampling rates of detection of driver characteristics, thereby allowing for introduction of further sensor arrangements without having to recalibrate the complete drowsiness detection system. Thus, it is therefore desirable to provide a method which allows for more flexibility in fusion of sensor data, further improving the robustness of drowsiness detection.
According to an aspect of the invention, the above is at least partly met by a method for determining an operational state of a driver of a vehicle using an awareness detection arrangement, the awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, the method comprising receiving, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle, comparing the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively, determining based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states, and weighing the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver.
The invention is based on the understanding that it is desirable to fuse several sources of information for making a better decision in relation to determining an operational state of a driver, but that prior art methods for fusion of information typically use optimization schemes to reach convergence for the multiple sources of information, resulting in that valuable information is lost in the process. Such optimization schemes may for example be based on the use of fixed rules to weigh the data into a single metric of driver drowsiness, neural networks, or various statistical methods to combine multiple indicators of drowsiness into a single output, which is used to issue a warning to the driver or activate certain countermeasures. A subsequent averaging or competition rule is then used to combine them into a single output. Moreover, existing solutions are often non-model based and require extensive validation.
According to the invention, physiological models of drowsiness evolution have been identified to be most useful when weighting the influence of different detectors in the final decision. Thus, it is of interest to predict the presence of a (truly) tired driver and to be able to warn in advance of a dangerous situation. The manifest drowsiness typically detected by driver state monitoring devices may be caused by both a physiological need for sleep and by fatigue caused by time-on-task or boredom. Thus, the manifest drowsiness may differ to varying degrees from the latent drowsiness actually caused by lack of sleep, in fact, there are a number of masking factors that temporarily may both increase or decrease the observed drowsiness level, such as external stimuli, long tedious road segments or even food intake. When such temporary masking factors are removed the latent drowsiness level will become dominant. In effect this may cause a person to appear fairly alert due to external stimuli, but quickly succumb to drowsiness once these masking factors are removed. Mathematical models of alertness/drowsiness can model the latent drowsiness, and when fused with the real time-monitoring of driver state, the accuracy and validity of the detection can increase significantly. Advantages includes for example improved predictive capabilities, less requirements for little customization and tuning for various implementations, resulting in a cost-effective implementation for achieving high robustness in detecting the operational state of the driver.
The inventive concept makes use of the output of multiple detectors (e.g. the first source of information, based on any sensor acting on virtually any time scale), and a predictive model of drowsiness (e.g. the second source of information and possible being a basic of a more advanced model), which is then treated using a known Bayesian approach to generate a robust multi-source indicator of drowsiness.
By means of the invention, it is possible to design for-the-purpose parameterized classes that are associated with specific actions to be triggered by separate in-vehicle systems. This means that the subsystem will be designed to carry out a critical function of these separate systems at the very core of its design, instead of trying to later define a mapping of this behavior based on a generic output of a drowsy driving detection system (e.g. based on the driver states; alert, drowsy). A specific example of this is how to design a driver state sensitive threshold that, will extend an Adaptive Cruise Control system (ACC) headway time gap for situations when the driver is drowsy. One typical way would be for the ACC system to use the output of a drowsy driving warning system, assign a specifically designed function that should determine if the measured drowsiness is significantly critical for the ACC to change its headway sensitivity or not. One would then subsequently create all necessary logic to handle potential errors or low-confidence of this data.
It should be noted that the wording “physiological data” in the following should be interpreted as all type of data that may be identified by an image based system that identifies e.g. the operator's eyes, face, body, as well as eye gaze direction, eyelid closures, or by measurement of the driver's heart rate, brain, activity, stress level, breathing, etc. Additionally, the concept of driver states and to determine such states is discussed below in relation to the detailed description of the invention.
According to an embodiment, the operational data of the driver comprises information relating to at least one of eye, face, head, arm and body motion of the operator. Such driver related, information may for example be generated by means of an image capturing device arranged within the vehicle compartment and overlooking the driver of the vehicle. Other types of sensors, generating relevant information for use in relation to the inventive concept may for example include heart rate sensors arranged in conjunction to the steering wheel or the driver seat. Additionally, movement sensors for generating indication of driver movement may be integrated within the driver seat and used for generating information useable in relation to the inventive concept.
According to a further embodiment, the operation of the vehicle may comprise information relating to at least one of time to line crossing, distance to a further vehicle travelling in front of said vehicle, steering and/or wheel operation patterns. Such vehicle related information may be generated by e.g. an image capturing device, radar equipment, or any other type of sensor used in vehicle operation.
In an embodiment, the model of the driver operating the vehicle comprises a first component relating to at least one of the time of the day and the operational time of the vehicle (e.g. time on task), and a second component relating to the drowsiness level of the driver. The second component relating to the drowsiness level of the driver is based on at least one of a model of sleep latency, time of day, time on task, a circadian rhythm, and a sleep/wake homeostatic process. An exemplary model of driver drowsiness is provided in WO09126071, by the applicant, which is incorporated by reference in its entirety.
Preferably, the outcome of inventive method, the determined state of the driver, is provided to a vehicle system configured to implement a vehicle control functionality, the vehicle system adjusting the vehicle control functionality based on the driver state. This may for example be implemented as discussed above in relation to the Adaptive Cruise Control system, and/or in relation to e.g. a forward collision warning (FCW) system as will be discussed further below in relation to the detailed description of the invention. Additionally, the determined state of the driver may be provided to a drowsy-driver detection system for generating a warning to the driver state indicate that the driver is drowsy.
According to another aspect of the invention there is provided a control system for determining an operational state of a driver of a vehicle, the control system comprising a control unit, the control unit connected to an awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, wherein the control unit is configured to receive, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle, compare the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively, determine based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states, and weigh the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver. This aspect of the invention, provides similar advantages as discussed above in relation to the previous aspect of the invention.
The control system may for example form pan of a vehicle system, further comprising the above disclosed awareness detection arrangement. Preferably, at least one of the first and the second source may be configured to generate operational data of the driver comprises information relating to at least one of eye, face, head, arm and body motion of the operator, where at least one of the first and the second source is an image capturing device. Additionally, at least one of the first and the second source may be configured to generate operational data of the vehicle comprises information relating to at least one of time to line crossing, distance to a further vehicle travelling in front of the vehicle, steering and/or wheel operation pattern.
According to a still further aspect of the invention, there is provided a computer readable medium embodying a computer program product for determining an operational state of a driver of a vehicle using an awareness detection arrangement, the awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, the computer program product comprising code configured to, when executed by a processor receive, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle, compare the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively, determine based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states, and weigh the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver. Also this aspect of the invention provides similar advantages as discussed above in relation to the previous aspects of the invention.
The processor may preferably be provided in a vehicle control unit, a computer, server or similarly, and the computer readable medium may be one of a removable nonvolatile random access memory, a hard disk drive, a floppy disk, a CD-ROM, a DVD-ROM, a USB memory, an SD memory card, or a similar computer readable medium known in the art (present and future). The present invention may be implemented using a combination of software and hardware elements.
Further features of, and advantages with, the present invention will become apparent when studying the appended claims and the following description. The skilled addressee realize that different features of the present invention may be combined to create embodiments other than those described in the following, without departing from the scope of the present invention.